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This report is the third of three reports examining SES 
within the Wake County Public School System 
(WCPSS). The first report, Supplemental Education 
Services: 2008-09 & 2009-10, provided a description 
of SES within WCPSS in terms of the schools, 
providers, and student participants in 2008-09 and 
2009-10 (Paeplow & Baenen, 2011). The second SES 
report, Implementation of Supplemental Education 
Services: 2009-10, examined the implementation of 
SES within WCPSS in 2009-10 (Paeplow, 2011). This 
report will focus on the long-term goal of improved 
student achievement by the end of the school year. 

Title I schools that have not made Adequate Yearly 
Progress (AYP) for three years enter into their second 
year of school improvement and are required to offer 
eligible students (i.e., economically disadvantaged 
students who receive free or reduced-price lunch 
[FRL]) SES in addition to existing Title I services. 

For additional information on AYP see ABCs and 
AYP results, WCPSS: 2009-10 (Haynie, 2011). 

Supplemental Education Services are designed to 
provide FRL students opportunities for academic 
instruction in addition to the instruction received 
during the school day. By offering FRL students 
additional opportunities for academic support, SES 
recognizes the economic constraints that may restrict 
these students from accessing extra learning 
opportunities in the same manner as students from 
more affluent families. Federal Title I legislation 
under the Title I, Part A of the Elementary and 
Secondary Education Act of 1965 as amended by the 
No Child Left Behind Act of 2001 (NCLB), focuses 
on improving the academic achievement of 

disadvantaged students; thus, students 
who receive FRL are eligible for SES 
regardless of their achievement level 
(DPI, 2008). 

WAKE COUNTY 

PUBLIC SCHOOL SYSTEM 



Major Findings 

Student Characteristics: In 2009-10, 508 
students participated in SES at the five schools 
offering SES: Brentwood, Fox Road, Poe, 
Wendell, and York. The vast majority of 
students (83%) participating in SES received 
tutoring services in both reading and 
mathematics. 

Implementation: Based on the prior two SES 
studies, SES was compliant with federal 
guidelines and services were implemented with 
fidelity with minor areas for improvement. 

Impact: Overall, this study did not support the 
value of SES in promoting higher achievement 
(beyond what matched students achieved). 

• Proficiency: The percentage of SES 
participants proficient in reading and 
mathematics in grades 4 and 5 and reading in 
grades 1 and 2 increased following services. 
However, participation in SES generally did 
not improve students’ academic achievement 
beyond that achieved by matched students 
except in grade 2 reading. 

• Growth: Considerable increases were 
evident between 2008-09 and 2009-10 in the 
percentage of SES students meeting state 
growth targets. The percentage of students 
meeting growth targets was greater than for 
all other reference groups in mathematics in 
2009-10. However, growth results were 
statistically similar to the matched group. 

Recommendations: All vendors should consider 
ways to strengthen their services. WCPSS staff 
should work with North Carolina’s Department 
of Public Instruction staff to determine if 
adjustments to current SES requirements are 
warranted. School Site Coordinators should 
consistently share information with classroom 
teachers, and communication between providers 
and classroom teachers should increase. 


Impact Evaluation reports provide basic 
evaluative outcome information on standard 
indicators. 


The author acknowledges the contribution of Nancy Baenen to this report. 
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SES is designed to improve student achievement by providing additional academic support to 
FRL students. In order to assess the success of this initiative, the long-term goals for 2009-10 
were examined: 

• A higher percentage of SES students scoring proficient on reading and mathematics End-of- 
Grade exams (EOGs) than the prior year before receiving SES. 

• A higher percentage of SES participants meeting reading and mathematics growth targets 
(academic change) than prior year before receiving SES (grades 4 and 5). 

• All subgroups meeting reading and mathematics growth targets (academic change) on EOG. 

Short-term and intermediate goals were set to improve the likelihood of reaching the long-term 
goal of improved student achievement (by the end of the year in which services were received). 
The first and second SES reports in this series examined the degree to which requirements were 
satisfied and short-term and intermediate goals were met. The first SES report, Supplemental 
Education Services: 2008-09 & 2009-10, found that the SES programs for WCPSS in 2008-09 
and 2009-10 were in compliance with federal guidelines to make available an SES program, 
enroll only FRL students, and to use state-approved vendors to deliver the program (Paeplow & 
Baenen, 2011). The second SES report, Implementation of Supplemental Education Services: 
2009-10, which examined the implementation of SES within WCPSS in 2009-10, found overall 
the SES program was largely implemented with fidelity (Paeplow, 201 1). Indeed, most of the 
Local Education Agency (LEA), provider, and parent requirements were met, with some areas 
for refinement related to communication and monitoring. This report will examine whether SES 
has improved student achievement and thereby met its long-term goal. 

STUDY DESIGN 

Student outcomes for schools offering SES in 2009-10 were examined and a comparison group 
of matched students was constructed from Title I schools not offering SES. The three key 
questions of interest in this evaluation were: 

• Has SES been effective in terms of outcomes for students (compared to students not served, 
FRL students not served, and the district overall)? 

• How did the students in the program progress compared to matched students? 

• How did the students in the program progress by NCLB subgroups? 

MATCHED SCHOOLS 

The five elementary schools that offered SES in 2009-10 were included in the study: Brentwood, 
Fox Road, Poe, Wendell, and York. Cluster analysis was conducted to select five Title I 
elementary schools from which to select a one-to-one matched group of students for comparison 
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to SES served students. The analysis was run using the centroid method: 1 three variables were 
included in the model (2008-09 performance composite, 2008-09 overall risk score, and 2008-09 
days in membership on the 20th day of school). The overall risk score is a school level score 
calculated based on the percentage of students at each school with academic risk factors, such as 
FRL status, limited English proficiency (LEP) status, and students with disabilities (SWD). 

Table 1 

Schools Offering SES and Matched Title I Elementary Schools 


SES School 

Matched School 

Brentwood 

Creech Road 

Fox Road 

Bar well Road 

Poe 

Smith 

Wendell 

Conn 

York 

Powell 


Data Source: 2008-09 Healthy Schools Indicators 
for Elementary Schools 


STUDY PARTICIPANTS 

The students who either attended one of the five schools offering SES or one of the five 
comparison schools in 2009-10 were included in this study. In 2009-10, 508 students 
participated in SES at one of the five schools offering SES. The vast majority (83%) of students 
participating in SES received tutoring services in both reading and mathematics. All students 
participating in SES were FRL as required; thus, a matched group of students was selected from 
the 1,678 FRL students attending one of the matched schools in 2009-10. 

MATCHED STUDENTS 

A one-to-one matched group of students was generated from schools offering SES and matched 
comparison schools. The one-to-one matching process between students participating in SES 
and FRL students attending a comparison school was run twice: once utilizing reading pre-scores 
and once utilizing mathematics pre-scores. Students were matched on grade level, their prior 
reading or mathematics performance, and several demographic characteristics — FRL, LEP, and 
SWD status. Students in kindergarten were not included in the reading or mathematics match 
process due to a lack of prior scores. For students in grades 4 and 5 their prior year’s score was 
their EOG score (+ or - three points or within the standard error of measurement for reading and 
mathematics 2 ). For students in grades 1, 2, and 3 the prior year’s mathematics score was based 
on the number of mathematics strands (algebra, number sense, geometry, data and analysis, and 

1 “Centroid method. The cluster to be merged is the one with the smallest sum of distances between cluster means 
(centroids) for all variables. The centroid method also weights for differences in cluster size” (Garson, 2010, p.8). 

2 With large-scale tests like the EOGs there is always some portion of the student’s score that can be attributed to 
unknown factors. The standard error of measurement provides an estimate for this unknown portion (Holdzkom, 
Sumner, & McMillen, 2010). The standard error of measurement for Spring 2007 to Spring 2010 was three in 
reading and four in mathematics for the grade 3 EOGs and three for reading and mathematics for the grade 4 EOGs. 
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measurement) in which they were proficient (i.e. scored level 3 or 4). For grades 1, 2, and 3 the 
reading prior year’s score was based on book level proficiency for the end of the prior grade 
level 3 . 

Of the 508 students included in the overall study, 463 received services in reading and 458 
received services in mathematics (most students received services in both subjects). SES 
participants were matched to FRL students attending the five Title I schools selected as 
comparisons; 367 of the 463 students served in reading and 389 of the 458 students served in 
mathematics had pre-scores available for matching. Thus, of the 367 students served in reading 
with available data, 211 were matched one-to-one based on grade level, reading pre-scores, and 
demographic characteristics; and of the 389 students served in mathematics with available data, 
214 were matched based on grade level, mathematics pre-scores, and demographic 
characteristics. 

The comparative analyses on the 422 one-to-one matched students for reading (211 SES students 
and 211 matched comparison students) and the 428 one-to-one matched students for mathematics 
(214 SES students and 214 matched comparison students) were conducted to verify the study’s 
findings. The one-to-one matched student comparisons were used to verify significant or notable 
findings. 


PROGRAM COSTS 

NCLB requires joint funding for the provision of school choice and SES in an amount equal to 
20% of its Title I, Part A allocation. Thus, as required by NCLB, in 2009-10 WCPSS set aside 
20% or $4,138,186.80 of the $20,690,934 received in Title I funding for the provision of school 
choice and SES. 

In 2009-10, $633,357.35 of the $4,138,186.80 set aside for the provision of school choice and 
SES was paid to SES providers. This amount funded tutoring services for 508 students attending 
one of the five Title I schools required to offer SES in 2009-10, at a cost of approximately 
$1,247 per student. The maximum per pupil allocation for SES in 2009-10 was $1,422. 4 Based 
on a total of 13,330 hours of tutoring received in 2009-10, the approximate cost was $47.50 an 
hour for the small group tutoring. Table 2 displays the 2009-10 SES expenditures by each of the 
provider agencies. 


3 Book Levels and mathematics strands are not based on standardized instruments; thus, they are less reliable than 
EOG assessments given to students in grades 3-5. Indeed, concerns about the reliability of the running records 
assessment in which the book level is derived have been raised locally and nationally (Blaiklock, 2003). 

4 In 2009-10, the per pupil cap was $1,422. “An LEA must calculate the per-pupil cap on SES costs by dividing its 
Title I, Part A allocation by the number of children residing within the LEA aged 5-17 who are from families below 
the poverty level, as determined by the most recent census estimates from the Department of Commerce” (U.S. 
Department of Education, 2009, p. 48). 
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Although $989,712 was set aside for the provision of SES, $633,357.35 was actually paid to SES 
providers. The difference is due to attendance rates since providers were paid based on actual 
tutoring hours provided. The largest expense in 2009-10 was paid to Sylvan Learning Center, 
the largest SES provider who served 155 or 30.5% of SES participants. 

Table 2 

2009-2010 SES Contractual Services 


Vendor 

Amount Ordered 

Amount Billed 

Academic Achievers/S &L Consultants 

$98,118.00 

$75,389.70 

Academics Plus, Inc. 

$118,026.00 

$103,450.50 

Accelerated Achievement/ Measurement Incorporated 

$98,118.00 

$70,886.70 

Achieve Success Tutoring by University Instructors 

$28,440.00 

$15,642.00 

AIM by Salient Learning 

$28,440.00 

$11,020.50 

Allied Academics 

$28,440.00 

$11,542.50 

ATS Project Success 

$21,330.00 

$4,266.00 

Capital Educational Supp. 

$21,330.00 

- 

It’s Simply English 

$21,330.00 

$3,981.60 

Learning & You Contractors, LLC 

$65,412.00 

$44,486.60 

Master Mind Prep Learning Solutions, Inc. 

$21,330.00 

$9,024.00 

Shaw University/Historically Minority Colleges & Univ. 
Consortium of NC 

$76,788.00 

$61,123.50 

Sterling Learning Center, Inc. 

$21,330.00 

$5,092.50 

Sylvan Learning Center / Ace It 

$319,950.00 

$200,387.25 

Village Learning Solutions 

$21,330.00 

$17,064.00 

Total 

$989,712.00 

$633,357.35 


Data Source: Oracle report provided by WCPSS’ State and Federal Programs staff. 


STUDENT CHARACTERISTICS 
SES PARTICIPANTS’ DEMOGRAPHICS 

Table 3 displays the demographic characteristics of the students participating in one of the five 
schools offering SES in 2009-10. Per the federal requirement, all 508 students participating in 
SES were FRL. 
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Among SES participants several subgroups of students were disproportionately represented. 

• LEP students represented 39.2% of served students compared to 12.2% in the district. 

• Black/ African American students represented 47.0% versus 23.7% of WCPSS students. 

• Hispanic/Latino students represented 39.2% versus 13.9% of WCPSS students. 


Table 3 

Characteristics of Students Participating in SES 
2009-10 



Students Participating 
in SES 

WCPSS Elementary 
Students 


Number 

Percent 

Number 

Percent 

FRL 

508 

100.0% 

22,966 

33.9% 

SWD 

69 

13.6% 

7,888 

1 1 .6% 

LEP 

199 

39.2% 

8,253 

12.2% 

Male 

259 

51 .0% 

34,417 

50.8% 

Female 

249 

49.0% 

33,331 

49.2% 

American Indian 

0 

0.0% 

172 

0.3% 

Asian 

11 

2.2% 

4,426 

6.5% 

Black/ African Am. 

239 

47.0% 

16,050 

23.7% 

Hispanic/Latino 

199 

39.2% 

9,393 

13.9% 

Multiracial 

23 

4.5% 

3,723 

5.5% 

White 

36 

7.1% 

33,984 

50.2% 

Total 

508 

100% 

67,748 

100% 


Note: Students will appear in more than one category: race and gender, FRL, SWD, and/or 

LEP. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters and WCPSS Demographics: School 
Statistics and Maps, 2009-10 at http://www.wcpss.net/demographics/reports/book09a.pdf 
Interpretation Example: Of the 508 students participating in SES 69 (13.6%) were SWD students. 


READING MATCHED GROUP DEMOGRAPHICS 

The one-to-one matched subgroups of students drawn from the study cohorts for reading and 
mathematics are depicted in Tables 4 and 5 respectively. While Table 3 displays the 
demographic characteristics of all students participating in SES in 2009-10, the matched reading 
and mathematics subgroups represent only SES participants and the comparison students that 
were matched one-to-one. Thus, while an individual SES student will be in the total group, he or 
she may or may not be in the reading and/or mathematics subgroups. 
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Of the 508 students participating in SES in 2009-10, 367 were served in reading and had pre- 
scores available for matching; 21 1 of these students were matched on their grade level, prior 
reading performance and several demographic characteristics — FRL, LEP, and SWD status — to 
students attending a matched school. Table 4 shows the demographic characteristics of the 21 1 
SES students and the one-to-one matched group of students. The SES and comparison student 
groups have exact representation for each of the matched variables. While students were not 
matched on gender or race, the distribution of these characteristics is very similar between these 
two student groups. 


Table 4 

Characteristics of Reading Matched Students 
in SES and Matched Students 



SES 

Matched Students 


Number 

Percent 

Number 

Percent 

FRL 

211 

100.0% 

211 

100.0% 

SWD 

23 

10.9% 

23 

10.9% 

LEP 

90 

42.7% 

90 

42.7% 

Male 

100 

47.4% 

106 

50.2% 

Female 

111 

52.6% 

105 

49.8% 

American Indian 

0 

0.0% 

0 

0.0% 

Asian 

6 

2.8% 

2 

0.9% 

Black/ African Am. 

98 

46.4% 

100 

47.4% 

Hispanic/Latino 

85 

40.3% 

92 

43.6% 

Multiracial 

11 

5.2% 

7 

3.3% 

White 

11 

5.2% 

10 

4.7% 

Total 

211 

100% 

211 

100% 


Note: 1. Students will appear in more than one category: race and gender, FRL, SWD, 

and/or LEP. 

2. Kindergarten students are not included in this table due to a lack of pre- scores 
for matching. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: Of the 211 students participating in SES matched one-to-one with a 

comparison student based on reading pre-score, FRL, SWD, and LEP status, 23 
(10.9%) were SWD students. 
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MATHEMATICS MATCHED GROUP DEMOGRAPHICS 

Two hundred and fourteen of the 389 students receiving SES services in mathematics in 2009-10 
with available pre-scores were matched on their grade level, prior mathematics performance, and 
several demographic characteristics — FRL, LEP, and SWD status — to students attending a 
matched school. Table 5 shows the demographic characteristics of the 214 SES students and the 
one-to-one matched group of students. While SES and comparison student groups have exact 
representation on matched variables, they also had similar racial and gender distributions. 

Table 5 

Characteristics of Mathematics Matched Students 
in SES and Comparison Group 



SES 

Comparison Students 


Number 

Percent 

Number 

Percent 

FRL 

214 

100.0% 

214 

100.0% 

SWD 

28 

13.1% 

28 

13.1% 

LEP 

86 

40.2% 

86 

40.2% 

Male 

102 

47.7% 

106 

49.5% 

Female 

112 

52.3% 

108 

50.5% 

American Indian 

0 

0.0% 

0 

0.0% 

Asian 

6 

2.8% 

4 

1 .9% 

Black/ African Am. 

100 

46.7% 

92 

43.0% 

Hispanic/Latino 

89 

41 .6% 

90 

42.1% 

Multiracial 

6 

2.8% 

9 

4.2% 

White 

13 

6.1% 

19 

8.9% 

Total 

214 

100% 

214 

100% 


Note: 1 . Students will appear in more than one category: race and gender, FRL, SWD, 

and/or LEP. 

2. Kindergarten students are not included in this table due to a lack of pre-scores 
for matching. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: Of the 214 students participating in SES matched one-to-one with a 

comparison student based on mathematics pre-score, FRL, SWD, and LEP status, 
28 (13.1%) were SWD students. 
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SES PARTICIPANTS’ GRADE LEVEL 

Figure 1 depicts students participating in SES in 2009-10 by grade level. While students from 
each grade level (K-5) participated in SES, over 40% of participants were in grades 1 and 2. 

Figure 1 

Students Participating in SES by Grade 
2009-10 



Data Source: 2009-10 End-of-Year Elementary School Student Rosters 


SCHOOLS OFFERING SES 

Among the 508 students receiving services at the five elementary schools offering SES in 
2009-10, 186 (36.6%) attended Fox Road, 150 (29.5%) attended Brentwood, 56 (11.0%) 
attended Poe, 66 (13.0%) attended Wendell, and 50 (9.8%) attended York (see Figure 2). 

Figure 2 

Students Participating in SES by School 
2009-10 



Data Source: Data file obtained from DPI via an electronic data file generated from the Consolidated 
Federal Data Collection System. 
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Question 1: Has SES been effective in terms of outcomes for students (compared to students not 
served, FRL students not served, and the district overall)? 

The 2009-10 reading and mathematics achievement results for all 
SES participants (K-5) are presented in Appendix A. In order to 
contextualize SES students’ 2009-10 academic achievement, the 
percentage of students proficient in 2008-09 was compared to the 
proficiency rate in 2009-10 for reading and mathematics. Only 
students with scores in both years were included within these 
comparisons. In reading, the percentage of SES participants 
proficient increased from 2008-09 to 2009-10. In mathematics, the 
percentage of participants proficient increased from 2008-09 to 
2009-10 among students in grades 4 and 5 and decreased among 
students in grades 1 and 2. 

Additionally, SES student performance is compared to non-SES participants attending one of the 
five schools offering SES in 2009-10, non-SES FRL students within these schools, and the 
district overall. 


In reading, the 
percentage of SES 
participants 
proficient increased 
slightly from the 
prior year. 

In mathematics, the 
percent proficient 
increased at grades 4 
and 5 and decreased 
at grades 1 and 2. 


GRADES 1 AND 2 

READING ACHIEVEMENT 2008-09 TO 2009-10 

Figure 3 depicts the percentage of students at or above grade level based on book level in 
2008-09 and 2009-10 for students enrolled in grades 1 and 2 in 2009-10 (kindergarten students 
are not included in this figure since they do not have 2008-09 data). First, it is important to note 
that over 60% of those served met the book level standard the year prior to service. Second, 
those served through SES showed slightly more positive changes than the other groups. 

• The percentage of students served by SES who performed at or above grade level based on 
book level increased slightly by 2.8 percentage points (from 61.3% to 64.1%). 

• The percentage of FRL students not served who were at or above grade level increased 1.8 
percentage points (from 66.1% to 67.9%), while for students not served and WCPSS overall 
the percentage of students at or above grade level remained relatively constant. 

Examined in another way, from 2008-09 to 2009-10, students participating in SES had the 
largest decrease in the number of students below grade level (55 to 51 students or 7.3%) 
compared to students not served (100 to 98 students or 2%), FRL students not served (74 to 70 
students or 5.4%), and WCPSS overall which actually experienced a slight increase in students 
scoring below grade level (2,670 to 2,802 students or 4.9%). 
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Figure 3 

Book Level 2008-09 and 2009-10, Grades 1 and 2 



Note: 1. Served = grade 1 and 2 students participating in SES in 2009-10; Not Served = grade 1 and 2 

students attending one of the five school offering SES in 2009-10 not participating in 
SES; FRL Not Served = students in the Not Served group who receive free or reduced- 
price lunch; and WCPSS = all grade 1 and 2 students with available data. 

2. Students with available 2008-09 and 2009-10 book level scores are shown in this figure. 

3. Groups are nested such that Served, Not Served, and FRL Not Served are included within 
WCPSS and FRL Not Served is a subgroup of Not Served. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: In 2008-09, 87 (61.3%) of served students were at or above grade level and in 2009- 
10 that number increased to 91 students (64.1%). 


MATHEMATICS PROFICIENCY 2008-09 TO 2009-10 

Figure 4 illustrates the percentage of SES students proficient in 2008-09 and 2009-10. For 
students in grades 1, 2, and 3 the prior year’s mathematics score was based on the number of 
mathematics strands (algebra, number sense, geometry, data and analysis, and measurement) in 
which they were proficient (i.e., scored level 3 or 4) in 2008-09; thus, for grade 1 students, prior 
year represents their score at the end of kindergarten. Most students in the SES program had 
shown mastery on four or five of the five math strands before service (70.6%). After service, 
Figure 4 illustrates that: 

• The percentage of students proficient in 4 or 5 mathematics strands decreased for both 
participants and matched non-participants. However, the drop was three times as large for 
those served (15.8 percentage points) versus those not served (4.9 percentage points). 
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• Decreases between 2008-09 and 2009-10 were also seen for FRL students not served and 
WCPSS students, but both decreases were considerably smaller than for those served. 

• SES participants had a larger percentage point increase (1 1.3 percentage points) in students 
who were either not proficient on any of the five mathematics strands or only proficient on 
one strand than students not served, FRL students not served, and WCPSS overall. 
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Figure 4 

Mathematics Strands 2008-09 and 2009-10 
Grades 1 and 2 
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Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: In 2008-09, 125 (70.6%) of served students were proficient on 4 or 5 
mathematics strands (number and operations; measurement; geometry; data 
analysis and probability; and algebra); in 2009-10 that number had decreased to 
91 students (54.8%). 


An additional analysis (not shown) found no differences in the patterns when disaggregated by 
initial mathematics achievement status. 
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GRADES 4 AND 5 

READING ACHIEVEMENT 2008-09 TO 2009-10 

Figure 5 displays the percentage of students in grades 4 and 5 at or above grade level (based on 
reading EOG performance) in 2008-09 and 2009-10. Grade 3 students are not included in this 
figure since they do not have a pre-score (i.e. 2008-09 EOG score). For all groups considered 
(SES served, not served attending a school offering SES, FRL not served attending a school 
offering SES, and WCPSS), the percentage of students at or above grade level increased between 
2008-09 and 2009-10. Students participating in SES had a smaller percentage point increase in 
students reaching grade level than FRL students not served (5.2 versus 7.0); the percentage point 
change was similar for those served versus all students not served (5.2 versus 5.6). Thus, the 
largest percentage point increase in students proficient was for FRL students attending one of the 
five schools offering SES who elected not to participate in the program. 

Figure 5 

Reading EOG 2008-09 and 2009-10 
Grades 4 and 5 



Note: 1. Served = grade 4 and 5 students participating in SES in 2009-10; Not Served = grade 4 

and 5 students attending a school offering SES in 2009-10 not participating in SES; 
FRL Not Served = students in the Not Served group who receive free or reduced-price 
lunch; and WCPSS = all grade 4 and 5 students with available data. 

2. Students with available 2008-09 and 2009-10 EOG scores are shown in this figure. 

3. Groups are nested such that Served, Not Served, and FRL Not Served are included 
within WCPSS and FRL Not Served is a subgroup of Not Served. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: In 2008-09, 54 (40.3%) of served students were at or above grade level and in 2009- 

10 that number increased to 61 students (45.5%). 
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MATHEMATICS ACHIEVEMENT 2008-09 TO 2009-10 

Similar to the reading results, both served and not served student groups made improvements in 
the percentage of students at or above grade level. Figure 6 shows the students served by SES 
had a larger increase in the percentage of students reaching grade level than those not served 
within their schools. SES students also experienced the largest reduction in those scoring below 
grade level (65 to 46 students, or a reduction of 29.2%) compared to students not served (138 to 
1 13 for students not served or 18.1%), and FRL students not served (106 to 88 students or 
17.0%). Thus, mathematics proficiency results showed more positive patterns than reading over 
time on the EOG. 


Figure 6 

Mathematics EOG 2008-09 and 2009-10 
Grades 4 and 5 



Note: 1. Served = grade 4 and 5 students participating in SES in 2009-10; Not Served = grade 4 

and 5 students attending one of the five school offering SES in 2009-10 not 
participating in SES; FRL Not Served = students in the Not Served group who receive 
free or reduced-price lunch; and WCPSS all grade 4 and 5 students with available data. 

2. Students with available 2008-09 and 2009-10 EOG scores are shown in this figure. 

3. Groups are nested such that Served, Not Served, and FRL Not Served are included 
within WCPSS and FRL Not Served is a subgroup of Not Served. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: In 2008-09, 65 (50.0%) of served students were at or above grade level and in 2009- 

10 that number increased to 84 students (64.6%). 
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ACADEMIC GROWTH 

Increasing the percentage of students reaching growth targets is 
another way to gauge success in improving achievement, and is 
more sensitive to student gains even when growth was not 
sufficient to change level scores. The state’s ABCs growth formula 
reflects approximately one year’s growth for one year of instruction 
for each student. Schools are considered to show high growth if 
60% of their students reach their growth target. 

In both reading and mathematics, the percentage of SES participants who met growth increased 
from 2008-09 to 2009-10 (see Figure 7). Increases were 16.6 percentage points in reading and 
26.7 percentage points in mathematics. Percent increases were considerable (48.5% in reading 
and more than 60% in math). In 2009-10, SES participants met high growth in mathematics, 
with 68.9% of students meeting their growth target. 

Figure 7 

Percentage of SES Participants Meeting 2008-09 and 2009-10 
Reading and Mathematics EOG Growth Targets 
Grades 4 and 5 


E&R Report No. 11.14 


A higher percentage 
of students met 
growth targets 
following SES 

services. 



Note: Students in membership 140 days or more are included in this figure. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 
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All student groups considered met growth for mathematics in 2009-10; but did not do so for 
reading (see Figure 8). The SES students served had slightly higher percentages of students 
reaching growth than the other three groups in mathematics. In reading, the percentage of 
students reaching growth targets was similar for SES students served, students not served, and 
FRL not served but was slightly lower than for WCPSS overall. While the student groups 
presented in Figure 8 were not matched to SES participants, considering how far below the 
district growth rate these students were before service (see Figure 7), these growth results are 
encouraging. 


Figure 8 

Percentage of Students Meeting 2009-10 
Reading and Mathematics EOG Growth Targets 
Grades 4 and 5 
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Note: Students in membership 140 days or more are included in this figure. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 


Question 2: How did the students in the program progress compared to matched students 
overall? 


Overall, the findings suggest that participation in SES generally did 
not improve students’ academic achievement beyond that achieved 
by matched students (measured by EOG proficiency and growth in 
grades 4 and 5 and book level and math strands in grades 1 and 2). 
There was however, one notable exception; among SES participants 
in grade 2 a significantly higher percentage reached proficiency in 
reading than did the matched students. 


Participation in SES 
generally did not 
improve students ’ 

academic 

achievement beyond 
that achieved by 
matched students. 

J 
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GRADES 1 AND 2 


READING ACHIEVEMENT 

Among students in grades 1 and 2 academic proficiency is compared for reading and 
mathematics (there is no growth component for the K-2 measures of proficiency). Students were 
matched on grade level, prior reading performance (i.e., book level proficiency for the end of the 
prior grade level), and several demographic characteristics — FRL, LEP, and SWD status. Figure 
9 shows the reading proficiency for SES participants and matched students in grades 1 and 2. In 
2009-10, a higher percentage of matched students were rated proficient than SES participants in 
grade 1; however, this difference was not significant. Among grade 2 students, a significantly 
higher percentage of SES students were proficient. 

Figure 9 

2009-10 Book Level for SES and Matched Students 
Grades 1 and 2 



Note: Significance based on a chi-square test for students participating in SES versus comparison 

students. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: Among served students in grade 2 in 2009-10, 42 (82.4%) were at or above grade 
level based on their reading book level compared to 30 (58.8%) of matched students. 
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MATHEMATICS ACHIEVEMENT 

Students were matched on grade level, prior mathematics performance, and several demographic 
characteristics — FRL, LEP, and SWD status. For students in grades 1, 2, and 3 the prior year’s 
mathematics score was based on the number of mathematics strands (algebra, number sense, 
geometry, data and analysis, and measurement) in which they were proficient (i.e. scored level 3 
or 4). Figure 10 displays the mathematics proficiency for SES participants and comparison 
students in grades 1, 2, and 3. While at each grade level a higher percentage of comparison 
students were proficient on 4 or 5 mathematics strands than were SES participants, the 
differences were not statistically significant. 

Figure 10 

2009-10 Mathematics Strands for SES and Matched Students 

Grades 1, 2, and 3 
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Note: Significance based on a chi-square test for students participating in SES versus 

comparison students. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: Among served students in grade 1, 35 (62.5 %) were at or above grade level on 
4 or 5 mathematics stands in 2009-10 compared to 42 (73.7%) of matched students. 
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GRADES 3 - 5 

EOG READING PROFICIENCY OF MATCHED STUDENTS 

The reading EOG proficiency levels for students who participated in SES and matched students 
are presented in Figure 11. The overall proficiency (Level III and IV) for the two groups was 
similar. While slightly more matched students (44.2%) scored a Level III or IV on their EOG 
than did SES students (41.6%), differences were not significant. Scale score differences were 
also not significant (see Appendix B). 


Figure 11 

2009-10 Reading EOG Level for 
SES and Matched Students, Grades 3-5 



Note: 1. n = students with an available 2009-10 Reading EOG score. 

2. Significance based on a chi-square test for students participating in SES versus 
comparison students. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: 38.5% of SES participants scored a Level III in 2009-10 compared to 40.0% of 
matched students. 
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EOG MATHEMATICS PROFICIENCY OF MATCHED STUDENTS 

Figure 12 depicts the mathematics EOG proficiency levels for students who participated in SES 
and matched students. A higher percentage of matched students (66.3%) scored a Level III or IV 
on their EOG than did SES students (58.1%); however, differences were not significant. Scale 
score differences were also not significant (see Appendix B). 


Figure 12 

2009-10 Mathematics EOG Level for 
SES and Comparison Students, Grades 3-5 



Note: 1. n = students with an available 2009-10 Mathematics EOG score. 

2. Significance based on a chi-square test for students participating in SES versus matched 
students. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: 52.0% of SES participants scored a Level III in 2009-10 compared to 60.2% of 
matched students. 


ACADEMIC GROWTH OF MATCHED STUDENTS 

As previously mentioned, increasing the percentage of students reaching growth targets is a more 
sensitive way to gauge success in improving achievement even when growth was not sufficient 
to change level scores. The state’s ABCs growth formula reflects approximately one year’s 
growth for one year of instruction for each student; and schools are considered to show high 
growth if 60% of their students reach their growth target. 
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Overall Growth 

A higher percentage of SES participants met growth in mathematics than did their matched 
counterparts while the reverse was true for reading. However, the differences were not 
significant (see Figure 13). 


Figure 13 

Percentage of Students in Grades 4 and 5 Meeting 2009-10 
Reading and Mathematics EOG Growth Targets 



Note: 1. Significance based on a chi-square test for SES participants versus matched students. 

2. Students in membership 140 days or more are included in this figure. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 


Academic Change 

Academic change is another way of consider students’ growth from 2008-09 to 2009-10. The 
state ABCs academic change score reflects whether students as a group grew more or less than 
the target projection for that year. A growth score of zero means the target was met exactly. 
Figures 14 and 15 utilize boxplots to depict the mean, median, and range of the academic change 
score for SES served students and matched students in grades 4 and 5 enrolled in 2009-10. The 
box represents the majority of student scores (25 th to 75 th percentile). The “whiskers,” or vertical 
lines, extending from the box, represent the range of scores, with the most extreme scores 
denoted by small boxes. The range of reading scores (denoted by the “whiskers”) was greater 
for matched students, while in mathematics the range was similar for the two groups. The 
academic change scores of SES participants were slightly higher than that of matched students. 
Within each box, the mean is signified by a plus sign and the median by a horizontal line in the 
middle of the box. The average academic change scores in reading and mathematics hovered 
close to zero, indicating performance close to what was expected (see Figure 14 and 15 
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respectively). Although there were slight differences in the mean academic change scores 
between matched students and SES students, the differences were not statistically significant . 5 
Thus, the results should be considered approximately the same. 


Figure 14 

Reading EOG Academic Change Score 
SES and Matched Students, Grades 4 and 5, 2009-10 



Matched Students 


SES Students 


Min 

-2.0 

-1.7 

Mean 

0.06 

-.09 

Max 

1.29 

1.17 

Std Dev 

0.60 

0.56 

N 

47 

47 


Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: SES participants had an average academic change score slightly less than zero 
(signified by plus sign on blue bar). 


Figure 15 

Mathematics EOG Academic Change Score 
SES and Matched Students, Grades 4 and 5, 2009-10 



Matched Students SES Students 


Min 

-1.2 

-.81 

Mean 

0.01 

0.21 

Max 

1.00 

1.39 

Std Dev 

0.53 

0.51 

N 

55 

54 


Data Source: 2009-10 End-of-Year Elementary School Student Rosters 


5 Significance based on a t-test on the difference of mean academic change score for students participating in SES 
versus matched students. 
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Question 3: How did the students in the program progress by 
NCLB subgroups? 

SES seemed to benefit Hispanic/Latino students in reading and 
mathematics, LEP students in reading, and Black/ African 
American and SWD students in mathematics the most. 
Hispanic/Latino participants experienced the most growth, meeting 
high growth (more than 60% of students having met their growth 
target) in reading and mathematics; however, the percentage who 
met growth was not significantly higher than other student groups. 


SES seemed to benefit 
Hispanic/Latino 
students in reading 
and mathematics, LEP 
students in reading, 
and Black/African 
American and SWD 
students in 

mathematics the most. 

J 


GROWTH BY NCLB SUBGROUPS 


To address the question, How did the students in the program progress by NCLB subgroups, the 
percentage of students meeting growth was examined. In order to provide context for SES 
participants’ results, SES students’ performance was compared to non-SES participants attending 
one of the five schools offering SES in 2009-10, non-SES FRL students within these schools, 
and the district overall. Analyses of matched students by subgroup were not appropriate due to 
small subgroup sizes and are therefore not shown. 
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READING 

Overall, 438 or 86.2% of the 508 students participating in SES in 2009-10 were either 
Black/ African American or Hispanic/Latino. Therefore, results by ethnicity include only these 
two groups; other subgroup sizes were too small for inclusion (less than 20). Figure 16 
illustrates the percentage of students who met growth in reading within these ethnic groups. SES 
participants (who were Black/ African American and those who were Hispanic/Latino) had a 
higher percentage meet growth than students within those ethnic groups who were not served by 
SES. 

• Hispanic/Latino students met high growth with greater than 60% of students meeting growth 
for SES participants and FRL students not served, but not for all students not served or 
WCPSS overall. 

• While Black/ African American students did not meet high growth, SES participants had a 
slightly higher percentage of students make growth than students not served or FRL students 
not served. 


Figure 16 

Percentage of Students in Grades 4 and 5 Meeting 2009-10 
Reading EOG Growth Target by Ethnicity 
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65.0% 


48.5% 46.6% 46.0% 


50.0% 


SES Not 
n=66 Served 

n=281 


FRLNot WCPSS 
Served N= 4,464 
n= 1 87 


Black/ 

African American 


59.2% 60.3% 


54.1% 


SES Not 
n=40 Served 

n= 1 52 


FRLNot WCPSS 
Served N=2,176 
n= 1 41 


Hispanic/ 

Latino 


Note: 1. Due to group sizes smaller than 20 students among students participating in SES, Asian, 

White, and Multi-Racial students were not included in this figure. 

2. Students in membership 140 days or more are included in this figure. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: 65.0% of the Hispanic/Latino students participating in SES met growth in 2009-10, 

compared to 59.2% of students attending an SES school but not served, 60.3% FRL not served at 
an SES school, and 54.1% in WCPSS overall. 
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Figure 17 illustrates the percentage of SES participants and the comparison students who met 
growth in reading by FRL, LEP, and SWD status. Among SES students, the LEP student group 
met high growth, but the FRL and the SWD student groups did not. 

• Among FRL students, the percentage of students who met growth was similar across the 
groups. 

• Among LEP students, SES participants had a higher percentage of students meet growth than 
the other student groups considered, but differences were not statistically significant. 

• Among SWD students, SES participants had a higher percentage of students meet growth 
than students not served, but not FRL students not served or WCPSS, although differences 
between SES participants and other student groups were not significant. 

Figure 17 

Percentage of Students in Grades 4 and 5 Meeting 2009-10 
Reading EOG Growth Target by Academic Risk Factor 
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Note: Students in membership 140 days or more are included in this figure. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: 66.7% of LEP students participating in SES met growth for reading in 2009-10, compared to 
60.0% of students not served, 60.8% of FRL students not served, and 52.2% in WCPSS overall. 


25 


Outcomes for SES 2009-10 


E&R Report No. 11.14 


MATHEMATICS 

Figure 1 8 illustrates the percentage students who met growth in mathematics by ethnicity for 
students participating in SES, non-SES participants attending one of the five schools offering 
SES in 2009-10, non-SES FRL students within these schools, and the district overall. The figure 
only includes Black/ African American and Hispanic/Latino students due to the small group sizes 
(<20 students) of the other ethnic groups. Black/ African American and Hispanic/Latino students 
met high growth for all comparison groups; comparisons within both ethnic groups favored those 
served in SES. Hispanic/Latino results for each comparison were more positive than for those of 
Black/ African American students. 


Figure 18 

Percentage of Students in Grades 4 and 5 Meeting 2009-10 
Mathematics EOG Growth Target by Ethnicity 
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Note: 1. Due to group sizes smaller than 20 students among students participating in SES, Asian, 

White, and Multi-Racial students were not included in this figure. 

2. Students in membership 140 days or more are included in this figure. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: 73.7% of the Hispanic/Latino students participating in SES met growth for 

mathematics in 2009-10, compared to 69.9% of students not served, 71.0% of FRL students not 
served, and 63.5% in WCPSS overall. 
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Figure 19 illustrates the percentage of students who met growth in mathematics by FRL, LEP, 
and SWD status for students participating in SES and the comparison groups. Among SES 
students, the SWD student group met high growth in mathematics while the LEP and FRL (all 
SES participants) groups did not. The SWD group showed the most positive pattern with a 
higher percentage of SES students reaching growth than any of the comparison groups, although 
differences between SES participants and other student subgroups were not statistically 
significant. 


Figure 19 

Percentage of Students in Grades 4 and 5 Meeting 2009-10 
Mathematics EOG Growth Target by Risk Factor 
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Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: 73.9% of SWD students participating in SES met growth for mathematics in 2009-10, 

compared to 67.1% of not served students attending a school offering SES, 64.3% FRL not served at an 
SES school, and 57.8% in WCPSS overall. 
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CONCLUSIONS 

The long-term goal of SES was to improve student achievement. In order to contextualize the 
SES results, comparisons to matched students and other reference groups were provided in this 
evaluation. However, it is also helpful to consider how successful SES was in meeting its goals 
independent of a comparison. Table 6 displays the status of SES goals independent of a 
comparison and compared to the results of students matched one-to-one on prior achievement 
and demographic characteristics. The degree to which SES met its three goals was examined for 
reading and mathematics separately. 

Independent of a comparison, the outcomes for students participating in SES were relatively 
positive. In reading, two of the three goals were met; a higher percentage of SES students were 
proficient in reading and met growth in 2009-10 than in 2008-09 prior to services. The third 
goal, all subgroups meeting growth, was partially met with two of the five student subgroups 
considered meeting high growth. In mathematics, one of the three goals was met, with a higher 
percentage of SES participants meeting growth in 2009-10 than in 2008-09. Two of the three 
goals were partially met, with a higher percentage of SES participants in grades 4 and 5 
proficient in 2009-10, but not in grades 1 and 2; and three of the five student subgroups 
considered meeting high growth. SES seemed to benefit Hispanic/Latino students in reading and 
mathematics, LEP students in reading, and Black/ African American and SWD students in 
mathematics although within subgroups differences between SES participants and comparison 
student groups were not significant. 


Table 6 

Status of 2009-10 SES Long-term Goals 


Long-Term Goal 

Subject 

Status Independent of 
Comparison 

Results Compared to 
Matched Students 

A higher percentage of SES students 
proficient on reading and mathematics 
EOGs than prior year before receiving 
SES. 

Reading 

Met 

A significantly higher 
percentage of grade 2 students 
were proficient; however, no 
significant differences in 
grades 1, 4, or 5 

Math 

Partially Met 

(Not Met at grades 1 & 2. 
Met at grades 4 & 5.) 

No significant difference 

A higher percentage of SES participants 
meeting reading and mathematics growth 
targets (academic change) than prior year 
before receiving SES (grades 4 and 5). 

Reading 

Met 

No significant difference 

Math 

Met 

No significant difference 

All subgroups meeting reading and 
mathematics growth targets (academic 
change) on EOG. 

Reading 

Partially Met 

(met with 2 out of 5 
subgroups) 

No significant difference 

Math 

Partially Met 

(met with 3 out of 5 
subgroups) 

Comparisons to matched 
group could not be conducted 
due to small subgroup sizes 


Data Source: Data file obtained from DPI via an electronic data file generated from the Consolidated Federal Data 
Collection System and 2009-10 End-of-Year Elementary School Student Rosters. 
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However, effectiveness of a program is always best assessed in comparison to progress of similar 
students. In the case of SES, overall outcomes were similar to matched students not receiving 
SES. Thus, the findings suggest that participation in SES generally did not improve students’ 
academic achievement beyond that achieved by matched students and non-participants through 
regular instruction and supplements normally available. While overall SES participants’ 
academic achievement did not improve beyond that achieved by matched students, it is notable 
that grade 2 students performed significantly better in reading than matched students. 

DISCUSSION 

This series of reports evaluating SES within WCPSS found SES to be in compliance with federal 
guidelines, implemented with fidelity in most ways, and to have some evidence of improved 
student achievement (Paeplow & Baenen, 2011; Paeplow, 2011). Relative to the report on the 
first SES program at Hodge Road Elementary, which only examined reading, reading results 
were similar with more positive findings at grades 1 and 2 than in grades 4 and 5 (Paeplow & 
Baenen, 2006). In fact, in grade 2 the percentage of students proficient in 2009-10 was 
significantly higher for SES participants than matched students. However, the fact that overall 
SES participants did not show significant improvement over matched students (matched by pre- 
score and demographics attending similar Title I schools) raises further questions. 

Why did the percentage of SES participants proficient on mathematics strands decrease more 
dramatically than for other groups considered? The percentage of WCPSS students proficient 
on the five mathematics strands decreased by grade level such that a lower percentage of grade 1 
students were proficient than kindergarten students. This decrease may be due to increased 
standards as the grade level increases. However, the fact that SES participants proficient on 
mathematics strands decreased more dramatically than for other groups considered may signify 
the need to better match tutoring services to the curriculum, provide increased differentiation for 
students, and/or provide innovative instruction to capture student interest. 

Did individualized student learning plans create enough differentiation in students’ learning 
experiences? In 2006, an evaluation conducted of the SES program offered at Hodge Road 
Elementary, the first WCPSS school required to offer SES, found the inappropriate use of 
remedial curriculum materials with FRL students who began the program at or above grade level 
(Paeplow & Baenen, 2006). The second SES report, Implementation of Supplemental Education 
Services: 2009-10, found that in 2009-10, tutoring services were based on individualized learning 
plans developed utilizing each student’s prior achievement (Paeplow, 2011). However, given 
SES student results were not significantly better than matched students, further differentiation 
may be warranted and/or more engaging materials. 

Are adjustments to state requirements needed? North Carolina’s Department of Public 
Instruction (DPI) approves providers and establishes requirements for the provision of SES. 
However, given the SES program was implemented with fidelity there may be a necessity to 
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review current SES requirements to determine if elements such as the number of hours of service 
required should be increased (at least for the lower achieving students), if there are any best 
instructional practices that should be added, or whether tutoring curriculums have a strong 
enough alignment to the state’s curriculum and enough variation to truly meet the needs of all 
students. One change that seems appropriate is to move the requirement that school follow up 
with parents when students are not attending regularly from the school to the provider. State 
guidance on how to make the most of small group instruction and to differentiate service may be 
warranted as well. 

In March 2011, WCPSS submitted an application to provide SES services in 201 1-12. WCPSS 
would provide targeted interventions and enrichment for individual students through direct 
instruction by certified teachers and utilize web technology (which will also be accessible from 
home for students with access to a computer). After-school sessions will be offered twice a 
week for 10 weeks. Student groups will consist of no more than six students and transportation 
will be provided by the district. The curriculum selected is Voyager Passport™, a research- 
based reading intervention program targeting students in grades K-5. It will be important to 
consider the level of materials to be used and the degree of differentiation that is possible with 
the materials given that participants’ prior achievement varied widely in 2009-10, with most 
students scoring at grade level prior to service. 

Is SES unnecessary? The possibility that SES is not the best utilization of funds could be raised 
based on these results. SES is a federal requirement, but sharing concerns with the appropriate 
authorities is recommended. Perhaps other services meet similar needs and therefore no greater 
results are obtained through SES. Perhaps FRL students scoring at grade level do not need this 
type of service, but could benefit from something else. An extensive review of the literature and 
other models for delivery of SES could shed more light on this question. 


30 



Outcomes for SES 2009-10 


E&R Report No. 11.14 


RECOMMENDATIONS 

While a higher percentage of SES participants met growth in 2009-10 than in 2008-09 prior to 
receiving services, generally the SES program in 2009-10 did not improve students’ academic 
achievement beyond that achieved by matched students (with the exception of grade 2 reading). 
While sample sizes were smaller than would be optimal, the value added by this expenditure of 
funds could not be demonstrated in our study. SES is required through Title I regulations, but 
we offer the following recommendations for improvement: 

• Improve the quality of services provided. All vendors should consider these findings and 
determine ways to strengthen their services. Increased differentiation of services may still be 
needed. Vendors, site coordinators, and Title I staff should monitor teachers’ delivery of 
service to ensure this occurs, and intervene when it does not. DPI might provide videos or 
other guidance on how to make the most of small group instruction and to differentiate 
service as well. 

• Work with DPI staff to examine SES program requirements. DPI should examine current 
SES requirements to determine if adjustments should be made (i.e., number of hours, 
instructional strategies, or tutoring curriculum). 

• Share concerns with DPI and the U.S. Department of Education about the focus and 
success of the current SES program. These concerns should be shared with the positive 
motivation of providing the most effective support possible for students. Ways to improve 
the program or alternatives to it should also be shared. 

Additional recommendations from the first two reports in this series included: 

SUPPLEMENTAL EDUCATION SERVICES: 2008-09 & 2009-10 

• Provide academic support to non-eligible students performing below grade level. 

• Increase participation in SES. 

• Develop or refine electronic data collection systems with unique student identifiers. 

IMPLEMENTATION OF SUPPLEMENTAL EDUCATION SERVICES: 2009-10 

• School Site Coordinators should share information with classroom teachers. 

• Increase collaboration and clarify and refine existing monitoring procedures. 

• Monitor providers to ensure that parents of students not attending are contacted. 

• Collect pre- and post assessment results centrally and post-assess program completers. 

In light of this report’s finding, we are not recommending increased participation in SES as it is 
currently structured until the recommendations based on implementation findings and the 2009- 
10 student outcomes are addressed. Further differentiation of tutoring services and consistent 
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sharing of information between stake holders including school site coordinators, classroom 
teachers, and providers should strengthen services provided to students. It is recommended that 
WCPSS program planners take into account the findings within this report series and build into 
our SES program greater differentiation of services and increased communication between 
stakeholders. For further information regarding recommendations for strengthening the 
implementation of SES see the previous two SES reports in this series: Supplemental Education 
Services: 2008-09 & 2009-10 (Paeplow & Baenen, 2011) and Implementation of Supplemental 
Education Services: 2009-10 (Paeplow, 2011). 


32 



Outcomes for SES 2009-10 


E&R Report No. 11.14 


REFERENCES 

Blaiklock, K. (2003, December). A critique of running records. Paper presented at the New 
Zealand Association for Research in Education/ Australian Association for Research in 
Education Conference, Auckland: New Zealand. 

Department of Public Instruction. (2008, July). Answers to FAQs regarding free tutoring or 
supplemental educational services (SES). Retrieved from 
http://www.ncpublicschools.org/docs/nclb/ses/faqs/sesfaqs.pdf 

Garson D. (2010). Cluster analysis. Raleigh, NC: North Carolina State University. Retrieved 
May 26, 2010, from http://faculty.chass.ncsu.edU/garson/PA765/cluster.htm#hierarchical 

Haynie, G. (2011). ABCs andAYP results: Wake County Public School System (WCPSS): 2009- 
10. Retrieved from http://www.wcpss.net/evaluation-research/reports/201 l/1020abc-ayp09- 
10.pdf 

Holdzkom, D., Sumner, B., & McMillen, B. (2010). Test scores and the standard error of 
measurement. Raleigh, NC: Wake County Public School System. Retrieved from 
http://www.wcpss.net/evaluation-research/reports/2010/1013sem.pdf 

Paeplow, C., & Baenen, N. (2006). Evaluation of supplemental educational services at Hodge 
Road elementary 2005-06. Raleigh, NC: Wake County Public School System. Retrieved 
from http://www.wcpss.net/evaluation-research/reports/2006/0609ses_hodge_revised.pdf 

Paeplow, C.. & Baenen, N. (2011). Supplemental education services: 2008-09 & 2009-10. 
Raleigh, NC: Wake County Public School System. Retrieved from 
http://www.wcpss.net/evaluation-research/reports/2011/1101ses09-10.pdf 

Paeplow, C. (2011). Implementation of supplemental education services: 2009-10. Raleigh, NC: 
Wake County Public School System. Retrieved from http://www.wcpss.net/evaluation- 
research/reports/201 1/1 105 ses_imple_2009-10.pdf 

U.S. Department of Education. (2009). No child left behind: Supplemental educational services 
non-re gulatory guidance. Retrieved January 20, 2011, from 
http://www2.ed.gov/policy/elsec/guid/suppsvcsguid.doc 


33 



Outcomes for SES 2009-10 


E&R Report No. 11.14 


APPENDIX A 

SES PARTICIPANTS’ READING AND MATHEMATICS ACHIEVEMENT 2009-10 

This appendix includes all students tested in 2009-10, regardless of whether they had a pretest 
available or not. This provides readers with a sense of results for the total group, but it does not 
allow a comparison to prior performance nor to similar subgroups. A higher percentage of SES 
students were proficient in mathematics than in reading at all grade levels with the exception of 
students in grade 2. 

Grades K-2 

Among K-2 students, kindergarten students had the highest percentage of students performing at 
or above grade level in reading and mathematics (see Figure A1 and A2 respectively). While 
this pattern was also found at the district level, the percentage of SES students proficient in 
reading and mathematics in 2009-10 was considerably lower than for WCPSS overall (e.g., SES 
reading percentages were 15.1 percentage points lower for kindergarten students, 28.8 for grade 
1, and 17.4 for grade 2). Both kindergarten and grade 1 students were proficient at a higher rate 
on all five mathematics strands (algebra, number sense, geometry, data and analysis, and 
measurement) than for reading (based on book level proficiency). For grade 2 students, 
however, the reverse was true. 


Figure A1 

SES Participants 2009-10 Book Level, Grades K-2 
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n= 74 

At or Above Grade Level 

75.0% 

53.9% 

67.6% 

■ Below Grade Level 

25.0% 

46.2% 

32.4% 


Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: In 2009-10, 75.0% of kindergarten students served were at or above 
grade level based on their book level. 


Across all mathematics strands, kindergarten students showed the highest percentage of students 
who were proficient (receiving a level 3 or 4). Ninety-one percent of the SES kindergarten 
students received a level 3 or 4 on the algebra and geometry strands. Students in grades 1 and 2 
experienced lower rates of proficiency. There was a similar pattern among WCPSS K-2 students 
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proficient on all five strands (see Table Al). Among students in grade 1, the strand with the 
highest percentage of students proficient was numbers and operations (72.5%). While among 
grade 2 students, the strand with the highest percentage proficient was geometry (60.6%). 

Figure A2 

SES Participants 2009-10 Mathematics Strands 
Grades K-2 



Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: In 2009-10, 54.6% of served grade 2 students received a 3 on the Algebra strand. 


Table Al 

WCPSS Reading and Mathematics Proficiency Percentages K-2 Data, 2009-10 



Grade 

2008-09 

2009-10 

Book Level Standards 

K 

88.9%* s 

90.1% 

1 

80.1% v 

^ 82.7% 

2 

83.7% 

^ 85.0% 

Mathematics All Strands 

K 

80.8% n 

83.8% 

1 

73.4% N 

74.8% 

2 

66.9% 

69.0% 


Note: 1. * indicates that 5 to 9.9% of the K-2 population had missing scores. 

2. Arrows indicate cohorts of students (e.g., students in grade 1 in 2008-09 and 2 in 2009-10). 
Data Source: 2009-10 Annual K-5 Assessment Data 
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Grades 3-5 

A larger percentage of students served by SES in 2009-10 were proficient in mathematics 
(60.9%) than in reading (43.1%) based on EOG scores in the spring of 2010 6 (see Figure A3). 
While this pattern was also found in WCPSS grades 3-5, the percentage of SES students 
proficient in reading and mathematics in 2009-10 was considerably lower than the district overall 
(31.4 percentage points lower in reading and 23.9 in mathematics). 

Figure A3 

SES Participants 2009-10 Reading and Mathematics EOG Levels, Grades 3-5 


5.2% 



Note: Reading reflects students who received SES in reading and Math reflects students 

receiving SES in mathematics. 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 

Interpretation Example: In 2009-10, 37.9% of served students scored level III on the reading EOG. 

Table A2 

WCPSS 2009-10 Reading and Mathematics EOG Levels, Grades 3-5 



Reading 

N= 33,796 

Mathematics 

A=33,936 

Level IV 

27.4% 

37.7% 

Level III 

47.1% 

47.1% 

Level II 

12.7% 

10.4% 

Level I 

12.8% 

4.8% 


Note: Frequency missing: Reading =155 and Mathematics =15 

Data Source: 2009-10 End-of-Year Elementary School Student Rosters 


6 Reading and mathematics are shown together; however, note this does not indicated levels are based on similar 
scales; furthermore, scales based on expert judgment — not equating. Additionally, renorming of reading and 
mathematics EOGs occurs independently and are preformed on different schedules. 
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APPENDIX B 

Reading and Mathematics EOG Mean Scale Scores for 
Matched Students 



Grade 


SES 

Matched 

Difference 

Significance 

Reading 

4 

(«= 23 ) 

2008-09 

331.5 

332.3 

-0.8 

not significant 

2009-10 

340.8 

341.6 

-0.8 

not significant 

5 

(n= 24 ) 

2008-09 

337.0 

337.8 

-0.8 

not significant 

2009-10 

342.2 

344.3 

-2.1 

not significant 

Mathematics 

4 

(«= 27 ) 

2008-09 

337.6 

338.0 

-0.4 

not significant 

2009-10 

346.5 

344.3 

2.2 

not significant 

5 

(«= 30 ) 

2008-09 

343.7 

345.4 

-1.7 

not significant 

2009-10 

349.0 

351.8 

-2.8 

not significant 


Note: 

Data Source: 


Table reflects only students with an EOG scale score for 2008-09 and 2009-10 on the 300 scale; thus, 
students tested on alternative EOGs are not included in this table. 

2009-10 End-of-Year Elementary School Student Rosters 
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